ad 


In summary we see three major points. First, 
source-~level debugging has primacy. All professionals in 
our survey want to "fix it in the language in which it 


broke" (2). The thought that goes into debugging depends on 


the source code to a very large degree. 


Second, regardless of whether IDFs are used or not, four 
tasks are regularly applied in the job of debugging. They 
include symptom location and classification, bug location, 
hypothesis generation and testing, and information gathering 
and selection. 


Third and finally, when debugging becomes complicated and 
difficult, or progress slows significantly» programmers will 
continue debugging by trying alternative tacks. In 
contrast, scientists and engineers currently will abandon 
the program as a method to solve their substantive problems. 
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CHAPTER 1 


BACKGROUND 


Starting early in 1980, a group of the Human Factors Project 
began to try to distinguish the characteristics of effective 
online documentation from ineffective material. After several 
discussions, the group realized that it was not making progress 
developing guidelines. One person thought one thing; another 
thought another. There were no independent criteria for 
evaluating the variety of ideas. 


The group began to understand that it did not know enough about 
why people use online documentation and what makes them 
comfortable. The question was not just what satisfies "people," 
but what satisfies different individuals and diverse 
associations. Material effective for one person with particular 
tasks, knowledge, capabilities, and organizational requirements 
may well not be useful for another. The problem of creating 
effective online documentation is interrelated with various 
aspects of the complex disciplines of human psychology and 
general systems theory. 


At this point, one member suggested that a committee gather 
empirical data on how people use online documentation to solve 
actual problems. The idea was not to survey users' general 
impressions of online documentation, but to investigate how they 
really got around particular difficulties. 


1.1 QUESTIONNAIRE DEVELOPMENT 


With help from the Documentation and Standards Project, the 
committee began work on a questionnaire, titled "Obtaining 
Assistance while Using Computers Interactively." (See Appendix 
B: Coordinator Packet.) The questionnaire asked each respondent 
to describe a particular situation in which he or she needed to 
obtain information about using an interactive computing system 
before the individual could complete his or her task. The goals 
of the survey were two: 


e To understand better requirements for interactive user 
assistance. 


e To understand better how to design Human Factors 
questionnaires. 


Based on experience and previous discussions, the committee 
members identified twelve sources of information that people 
might use to solve problems which they encountered in using 
interactive computing systems. These sources ranged from online 
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help on the syntax of a command or help on an interactive process 
to written documentation or another person. 


The questionnaire asked the user to rate each of the sources that 
he or she had consulted in terms of their accessibility, 
understandability, and usefulness. As it ultimately evolved, the 
questionnaire also requested background information (on the type 
of work the respondent performed and experience with computing) 
and an overall appraisal of the information sources provided by 
the interactive system. There were fourteen questions total (see 
Appendix A.) 


The questionnaire was pre-tested twice, the first time at 
Stanford Linear Accelerator Center and the second time at 
Stanford University. As a result of this work, the order of the 
inquiry was changed, a couple of questions were dropped, and most 
were modified. Instances of the changes include: 


e The question asking for a description of a particular situation 
requiring information was moved earlier in the questionnaire 
(to number 4). Some people stopped responding when they 
encountered that question, even though the introduction stated 
that this was the central part of the survey. 


e Questions on what sources were available, as distinguished from 
what sources were used, were dropped. Respondents got confused 
having to refer back to how they had answered the availability 
and use questions, so instead a "Didn't use" field was added to 
the questions on the accessibility, understandability, and 
usefulness of the information obtained (numbers 8, 9, and 10). 


e A question asking people to estimate the number of times they 
had previously done the procedure for which they now needed 
information was dropped. A question asking people to rate 
their experience with the system on which they encountered the 
problem replaced it (number 7). Committee members thought it 
would be an alternative way of obtaining a measure of the 
users' relevant computing knowledge. 


e Much tinkering with the phraseology of the questionnaire was 
performed. The emphasis was changed from "problem" to 
"Situation requiring information." The terminology in the 
accessibility, understandability, and usefulness questions was 
also modified. For example, the question "Was the information 
you received relevant?" was changed to "The information I 
received was useful when I ..." (number 10). 


Finally, a question asking people to evaluate the overall quality 
of assistance on their interactive systems was inserted (number 
12) since respondents wrote their general appraisals anyway. 


The committee realized that many installations have added to or 
extended the facilities for interactive assistance provided by 
vendors, so the group developed a second questionnaire, "System 
Tailoring Information Questionnaire: Interactive User 
Assistance," to be filled out by one person at each installation. 


It was hoped that data from this questionnaire could be used to 
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distinguish the effectiveness of vendor-supplied assistance from 
installation-supplied facilities. For example, Cornell 
University has developed its own VM HELP command that allows for 


more flexible extraction of information than does IBM's. 


So far, examination of these data has not been attempted. Given 
what analyses of the user questionnaires have shown and not 
shown, it is unclear whether study of the installation-tailoring 
questionnaire would be useful. 


1.2 QUESTIONNAIRE ADMINISTRATION 


At several SHARE meetings, the committee recruited people (called 
installation coordinators) who agreed to distribute the user 
questionnaire to people in their organizations and to fill out 
the installation questionnaire. The Human Factors Project also 
obtained SHARE Board of Directors and Advisory Council 
authorization for a couple of vendor volunteers to recruit 
questionnaire coordinators from places using their products. 


The questionnaires were to be filled out anonymously. 
Maintaining installation anonymity was very important to some 
sites. (Those installations mentioned in this paper have given 
their permission to be explicitly cited.) 


In the spring of 1981, the questionnaire packets were sent to 
forty coordinators and the cooperating vendors. Twenty-eight 
coordinators (including those who volunteered in response to the 
vendors' requests) returned 270 user questionnaires, of which 229 
were usable. Most, if not all, of the respondents volunteered to 
fill out the questionnaire; therefore, the sample was not 
random. 


1.3 DATA ANALYSES 


One very important conclusion from the first round of analysis 
(Winters, 1983) was that distinct user problem states are 
identifiable. Responses to question 4 were categorized based on 
the experimenter's evaluation of the problem description by the 
respondent. The user problem state reflects the user's cognitive 
experience with that (specific) situation. 


Learning means the respondent has to learn at least some new 
concept, relationship, and/or nomenclature in order to work 
through the situation. For example, a FORTRAN programmer who is 
expert in MVS/TSO starts to learn VM/CMS, a PL/I programmer has 
to learn how to call SORT, or a secretary has to learn how to 
LOGON and SCRIPT a letter. 


Problem solving means the respondent knows all the critical 
concepts, relationships, and nomenclature but does not know how 
they fall together to resolve the situation. For example, a 
statistician needs to learn how to save recoded variables in 
SPSS, or a FORTRAN programmer has to figure out how to write a 
program to do list processing. 
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Refresh (specific) means the respondent has resolved a similar 
Situation in the past but needs some refreshing on specific 
information that has been forgotten. For example, a user needs 
to consult notes on how to send a message to another users, or a 
user requests online HELP for command syntax, e.g., option name. 


Refresh (qeneral) means the respondent needs refreshing on what 
he or she has forgotten for a situation which he or she describes 
at a general level. In summarizing the problem, the respondent 
does not include a specific goal. For example, a user consults 
notes about SPIRES, or a user rereads documentation on a mail 
facility. 


Some of the conclusions from the preliminary analyses using these 
problem states were: 


e Users in the learning state tended to use message details and 
command help more than users in other problem states. 


e Learners tended to talk to a consultant/supervisor to a greater 
degree than users with more cognitive experience. 


® User problem states are inversely related to the time to find 
information. 


The conclusion that learners require interactions with other 
humans is similar to that of a recent study undertaken at Boeing 
Computer Services (Marks, 1982). 


Other preliminary conclusions were: 


e User problem states are a significant element in designing an 
information source. 


e Most nonhuman information sources provide little satisfaction 
to users in the learning and problem solving states. 


e Most nonhuman information sources are applicable mainly to 
users requiring refresh information. 


e Today, the only consistently effective source of information 
for users in the problem-solving states is a human (i.e., 
another user or a consultant or supervisor.) 


® Nonhuman information sources will not be effective in learning 
and problem-solving states until all of the major elements of 
person-to-person dialogue are identified and incorporated into 
the structure of the information source. 


Providing effective interactive assistance through online 
documentation is but one aspect of dialogue design. Not much has 
yet been written on this subject. To aid those who would like to 
investigate the area further, other work that has come to the 
attention of the authors is noted in the Bibliography. 
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CHAPTER 2 


RESULTS 


The preliminary analyses concluded that a classification scheme 
for user problem states that seems to hold the most promise for 
analysis of results from this survey and for future research 
projects (see Section 1.3). Therefore, this classification 
formed the core of the second round of analyses included in this 
final report. 


2.1 EXAMINING THE RESPONDENTS 


Sections 2.2 through 2.8 repeat some of the tables shown at SHARE 
60, but are included here for sake of completeness. Before 
making any conclusions about user reaction to online and other 
problem aids, one should understand who the respondents are, 
where they come from, what is their experience, etc. 


Chapter 2: Results 7 


2.2 Business Environment 


Figure 2.2 Primary business of institution 


Business services II 2 


2 ----+ 
Computer and DP I 7 1 


4 He ee + 
Education I 81 I 


§ --- a eee + 
Finance I 39 I 


6 ----------- + 
Government I 21 1 


7] ------------------------- + 
Manufacturing I 50 I 


LO: + 
Transportation II 1 


1] ------ + 
Utilities I 111 


2 Ser SSSs= + 
Other I ae. 


VALID CASES 229 


Given the manner in which the survey questionnaire was 
distributed and administered and the particular energies of some 
of the sponsors connected with universities, it isn't surprising 
that the largest category of respondents come from education 
(35.4% from Figure 2.2.) However, the mix of other backgrounds 
is notable, particularly from finance (17.0%), government (9.2%), 
and manufacturing (21.8%.) 
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2.3 Job Environment 


Figure 2.3 Area job most related 


Administrative I 18 I 


2 meron nt 
Operations I 12 I 


3 ------- + 
Customer serv I 11 I 


4 -------~--------------- + 
Education I 33 I 


5 ----- + 
Finance I 7 I 


6 ---+ 
Management I I1-5 


7 <-+ 
Marketing II 1 


8 -----------------— + + + + + + + 


Programming I 


ee ae me eee Sem oe cane Ee Me cme Se nee me ume SD ume cme ee eae cm gems enter Gm eee ED me fee OD wee meee ee cee es eee ee ee te em ee ee ee ee cee ee 


Q ---------------------- + 
Research I 33 I 


10 ~----------------------- + 
Other I 36 I 


VALID CASES 229 


While Figure 2.2 shows that the largest business category of 
respondents is education, only 33 out of the 229 total indicate 
education as their job category (Figure 2.3.) The largest number 
of respondents is programmers (31.9%), followed by education and 
research (14.4% each), and administration (7.9%.) 
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2.4 Computer Experience 


Figure 2.4 Time using computers 


~+ 
< 1 month II 2 
—+ 
I 
2 ----- + 
1 to 6 mos I 8 I 
----- + 
I 
3S SSSsS5 ene - S55 + 
6 to 24 mos I 28 I 
a a ee ee ee eee ee oe ee oe + 
I 
alae eal et et maleate lane eal teeta teeteatentnateatententeetentententeteteeatenteatend + 
2 to 5 yrs I 52 1 
I I Nee ee + 
I 
Gee + 
5 to 10 yrs I 68 I 
om ee cee ee eee cee sees ee ee ee et ae nee Oe ee me ee ree ee ee eee ee ee ee eee ee ee ee Se ce ae ee Gs el ee ee oe ee oe + 
I 
© aasteesheatinetoestoestenstenstanstenteantenteteiaatestestentonteatestententententeatentantententetentetentententententantenentadentententen + 
10 or more yrs I 69 I 
eee aes cae ewer see me cee GN rey seen SE Gmte wees SED SOE: we Game ME SE GEE Oe oe meme GOUD cS que ome ORD GE tee GD te eee Ges Gi one Cu GED owe us aD we cue cafe 
VALID CASES 227 MISSING CASES 2 


oe ee ee eee ee ee cee ee ee ome ee me ce ee cee ee ee ee ome ee ee ee ee cee ee cme cme cre ee me re ae me ee ee oe ee ce wee ee Ore ce ee te ee eee eee ee me ee ee ee ee one ee ee ce ee ee 


It 1s easy to see from Figure 2.4 that this survey was conducted 
among very experienced computer users. Over 80% of the 
respondents have been using computers for at least two years, 
nearly 60% better than five years, and 30% better than ten years. 
Compare these results to Figure 2.6 showing the amount of time 
respondents has spent on the particular system they evaluated. 
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2.5 Computer System 


Figure 2.5 Interactive system used 


SPEAKEASY 


SPF 


6 
SPIRES 


TSO 


WYLBUR 


10 
Other 


11 
Don't know 


VALID CASES 229 


ne eee eee re Se ee ee ee ee ee eee ee ee ee cee cee me eee ee ame me ome Ore ome ame eee em ee ee ee 


ee ee ee ee ree ee eee ee ee we cee ee me me em ee ee ee ee ee ce ee ee eee te oe ee ee 


oe gee me cme Cm ee cen awe wee eum Gee mee cee mem cee cee seme ge me eee eee cme ee em mee ce ee te ee eee ee ee ee oe ee ee es ee ee ee 


There was never any intention to evaluate any particular system 
and there are not sufficient respondents in any single category 


for such detailed analyses. 


However, conclusions made regarding 


the use of online assistance, documentation, etc., should be made 
considering the mix of systems respresented. 


A crosstabulation of system with business 


spread of the systems 
shown in Figure 2.2. 


shown in Figure 2.5 
Excepting the small 


SPIRES, no system is represented in fewer 
categories and no business category shows i 
systems. Therefore, contamination by a single system in a single 


shows a reasonable 
across the businesses 
categories, CICS and 
than four business 
fewer than three 
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the two variables seem to be independent. ) 


Figure 2.6 Time using 


2.6 Amount of Time on the System 


system 


ll 


business category does not seem to be happening (in other words, 


< 1 month 


2 
1 to 6 mos 


3 
6 to 24 mos 


4 

2 to 5 yrs 
5 

5 to 10 yrs 
6 


10 or more yrs 


VALID CASES 229 


In contrast to Figure 2.4 displaying the relative time 
respondents have been uSing computers, Figure 2.6 shows that the 


bulk of respondents (38.4%) have had between one-half to two 
years' experience with the particular system. 
time for nearly anyone to become comfortable with a system. 

However, see Figure 2.7 for the respondents' self-evaluations. 


This is enough 
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2.7 Experience on the System 


Figure 2.7 How experienced are you on system 


ee eee ee + 
Not very I 59 I 
ee + 
I 
2 errr rrr rrr + 
Moderate I 107 I 
eee Sa 
I 
3 ---- ee + 
Very I 63 I 
ee ee ee ee ee + 
VALID CASES 229 MISSING CASES 0 


Time spent on a system (Figure 2.6) is not necessarily the best 
measure of one's familiarity because use may be sporatic or the 
tasks performed may tap only a limited range of the system's 
capabilities. Thus, the respondents were asked to rate 
themselves according to experience and nearly half of them 
(46.7%) describe themselves as "moderately" experienced. 


2.8 Description of the Problem 


Figure 2.8a Problem state 


tee ee eee ee ee re ee ee ee ee ee + 
Learn I Si. 
a ee a a i oe + 
I 
Q wr rr rr rrr rrr rrr rrr + 
Solve I 114 I 
i ee }- 
I 
3 ------ + 
Refresh (specific) I 22 I 
------ + 
I 
Qo me ------- + 
Refresh (general) I 37 I 
ee + 
VALID CASES 224 MISSING CASES 5 


As described in Section 1.3, the first round of analyses of these 
results produced a classification of the problem respondents 
described into four problem states. The breakdown of how 
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respondents fell into these four categories is shown in Figure 
2.8a. x 


Figure 2.8b Problem state in three categories 


ee ee oy + 
Learn I 51 7 
ee ee Oe ee eee ee ee ee ee ee ee ee + 
I 
Q mr rrr + 
Solve I 114 I 
ne ee > 
I 
3 --------------- + 
Refresh I 59: 
oe ae eee ae ee See Se eS Oe ome oe on oe oe + 
VALID CASES 224 MISSING CASES 3) 


oe ee ame ee ee ete cates ee cme em wee nm come SOE ee eee ee ce we eee cee ee mee ee ee ee cee mee ee See ee ce ee ee mee ne oe eee Oe me ee ee Se ee ees cee ee ee ee ee ee eee ee ee ee oe 


However, this round of analysis produced two observations: the 
opinions of those respondents whose problems fall into either 
"refresh" state are very Similar and there are insufficient 
numbers of respondents in either category. Therefore, since there 
seem to be good theoretical and statistical reasons for combining 
the two categories into a single refresh state, the remainder of 
the analyses use the classification shown in Figure 2.8b. 


2.9 EXAMINING RESULTS BY DEMOGRAPHICS 


The next step in the analysis is to examine whether respondents 
use or don't use the particular information source according to 
the various demographic breakdowns, whether or not they like what 
they use. Therefore, "Use" is defined has having indicated any 
of the agree-disagree choices in questions 8, 9, and 10 on the 
accessibility, understandability, or usefulness of the 
information source. For the special fourth "overall quality" 
attribute, use translates to availability of the source and non- 
use translates to unavailability. 


The methodology is to simply crosstabulate the 48 opinion 
variables (12 information sources times the four attributes of 
accessibility, understandability, usefulness, and overall 
quality.) The 48 variables are named with prefixes 


EASY ease of access to information 

UNDER understandability of information source 
USEFUL usefulness of information source 

QUAL overall quality of information source 


Te 
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for the four attributes. Each individual question is then noted 
by the numbers 1 through 12 as the suffix for the variable name. 
The sources are 
1 system offered suggestions 
system provides detail on message 
system outlines current situation and options 
system gives help on command 
system gives help on subject area 
system gives example 
tutorial at terminal 


listed resource index at terminal 


read document from terminal 


Oo wo wo NI WwW OH Fe WW KN 


a) 


read printed document 


~ 
i) 


talked to another user 


12° talked to a consultant or supervisor 


Thus variable EASYl1 is a respondent's answer to the ease of 


access question about the system offering suggestions. 


The previous research (Winters, 1983) showed strong correlations 


among accessibility, understandability, and usefulness. 
Therefore, whenever reportable results arise in more than one of 
these three attributes, only one is shown for brevity. Since the 
intention was that overall quality of the source (not necessarily 
in the context of the specific problem) be a measure independent 
of the success or non-success of problem solving, reportable 
results for that measure are also shown. 


2.10 Business 


In examining the respondents' business areas, two categories are 
eliminated because of limited numbers of cases (Business service 
and Transportation) along with the Other category. 


Tables 2.10a through 2.10e represent the three questions with 
reportable results although variables EASY4, UNDER4, UNDER7, and 
USEFUL7 are also reportable. Tables producing a chi-square 
Significance less than 0.02 are considered reportable. 
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Figure 2.10a System offered suggestions - Accessibility 



























































EASY1 
COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
BUSINESS’ -------- aS eee Pianeta rota + 
2 2 4 6 
Computer and DP 33:63 66.7 36a 
+-—-- + - ¢ - + 
4 35 38 73 
Education 47.9 52.1 38.2 
+-------- to-- + 
5 17 19 36 
Finance 47.2 52.8 18.8 
+-------- +-------- + 
6 5 16 21 
Government 23.8 76.2 11.0 
+—-—---—---- +-------- + 
7 13 32 45 
Manufacturing 28.9 71.1 23.6 
+-------- +-------- + 
11 8 2 10 
Utilities 80.0 20.0 542 
+-—------- +-------- + 
COLUMN 80 111 191 
TOTAL 41.9 58.1 100.0 
CHI-SQUARE Deeks SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
13.61236 5 0.0183 25513 3 OF 12 ( 25.0%) 
NUMBER OF MISSING OBSERVATIONS = 38 


eS ee ee me oe ce cee ee meee ee ree cee eee eee ee ee em ee ome ce ee Me SS Sree Gee Seem Gime GE im eee Ge Ge ms ee tie win mes eine em tee mh ee Meuith nie rm SD Cm nme rum rm GD omni cme eee im me cee ce cams ees ees 


Table 2.10a is statistically reportable, but it doesn't really 
reveal much about the pattern of usage of system suggestions 
among the various business categories. Likewise, Table 2.10b 
doesn't show a very remarkable pattern of differences among the 
business categories. However, comparison of the two tables shows 
that some respondents have access to this particular information 
source but don't use it. 
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Figure 2.10b System offered suggestions - Quality 



























































QUAL1 
. COUNT 
ROW PCT Don't Have ROW 
Have TOTAL 
BUSINESS =  -------- tome teo------ + 
2 1 6 7 
Computer and DP 14,3 85.7 3.8 
+---—----- +-------- + 
4 25 44 69 
Education 36.2 63.8 37.9 
+-------- +—-------- + 
5 8 25 33 
Finance 24.2 75.8 £6 od 
+-------- +—-—-—---~- + 
6 3 15 18 
Government 16.7 83.3 9.9 
+-------- +-------- + 
7 6 39 45 
Manufacturing i i Sorc: 86.7 24.7 
+-------- +-------- + 
i 7 3 10 
Utilities 70.0 30.0 5.5 
+--~------ +-------- + 
COLUMN 50 132 182 
TOTAL 27.5 1265 100.0 
CHI-SQUARE Dik: SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
18.08750 5 0.0028 1.923 3 OF 12 ( 25.0%) 
NUMBER OF MISSING OBSERVATIONS = 47 
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Figure 2.10c System gives help on command 



























































USEFUL4 
COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
BUSINESS) = -------- + ------ ea ama a + 
2 1 5 6 
Computer and DP 16.7 83.3 3.5 
+-------- +-------- + 
4 23 43 66 
Education 34.8 65.2 38.4 
+-------- +-------- + 
5 22 10 32 
Finance 68.8 31.3 18.6 
S eteeteniestetestenton +-~------- + 
6 5 16 21 
Government 23.8 76.2 LZs2 
+--—------ +-------- + 
7 15 23 38 
Manufacturing 39.5 60.5 2263 
+-~------ +-------- + 
11 2 7 9 
Utilities — 22:2 77.8 542 
+-------- +-------- + 
COLUMN 6 104 172 
TOTAL 39.5 60.5 100.0 
CHI-SQUARE DF. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
16.64547 5 0.0052 2eo%2 3 OF 12 ( 25.0%) 
NUMBER OF MISSING OBSERVATIONS = 57 


Figure 2.10c shows that most respondents tend to use a system's 
ability to provide help on command, with the exception of those 
in the Finance category. Again, these results probabiy offer no 
substantive conclusions. 
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Figure 2.10d Tutorial at terminal - Accessibility 
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EASY7 
COUNT 
ROW PCT Don't Use ROW 
3 Use TOTAL 
BUSINESS pe ee eee pe ee eet 
. 2 2 4 6 
Computer and DP | 33.3 | 66.7 3.2 
a 
ee eet & 4 53 18 y 2 ne 
Education | 74.6 —625.4 38.0 
. ee ie in ee swt es ate op 
mee 7 5 28” = 16 36 
Finance 55.6 44,4 19.3 
| Sl eslanlententutententante shenteetententententeaton + 
. . 6 —«B 13 21 
Government 38.1 61.9 elg'2 
De eo 
7 17 27 44 
Manufacturing 38.6 61.4 23.5 
: siestedienteateatenbentenn secteteatesteatettetonte 5 
11 7 2 9 
Utilities 77.8 2242 4.8 
$22 + + 
COLUMN 10 80 187 
TOTAL 57.2 42.8 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
21.14811 5 0.0008 2.567 3 OF 12 ( 25.03%) 
NUMBER OF MISSING OBSERVATIONS = 42 


a i ean ih Ont caine eatin mam Cer cube sine Wilh Gini GUNNER Gantt Orne TY GD itty ULI CET tiem mnt Gitte Gai LN GUI nti Gem GEES GO GUND GHOSE efums Gimme GEE GmTED GUE GRD GONE eruiee Om DtAWy GY Gmc ovine GOEL ime Vimy Stk Si GaN UAH GHTRY mw Genin kD HNN GAO GID Ge lite 


Figures 2.10d and 2.10e individually are not very revealing, 
except to note that the respondents in education seem to have no 
access to tutorials at the terminal, a remarkable conclusion 
considering the supposed role of tutorials in education. 

However, a comparison of the two figures again reveals the 
pattern of respondents having access to an information source but 
not using it. 
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Figure 2.10e Tutorial at terminal - Quality 
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QUAL7 
COUNT 
ROW PCT Don't Have ROW 
Have TOTAL 
BUSINESS Sen ee eee pertain oe a apes ee ie, ee + 
2 2 4 6 
Computer and DP 33.3 66.7 3.4 
$e eee Skeets + 
4 43 23 66 
Education 65.2 34.8 37.7 
| sentsteetententententoen t-—------- + 
5 6 29 35 
Finance 17.1 82.9 20.0 
S sloateeteeteeeleneton +-------- + 
6 2 15 17 
Government 11.8 ~—6BB.2 9.7 
S oeenieateetentententeet +---- = + 
7 8 34 42 
Manufacturing 19:0 81.0 24.0 
+—-------- +-------- + 
11 4 5 9 
Utilities 44.4 55.6 5.1 
+—-—------ $e + 
COLUMN 65 110 175 
TOTAL 37.1 62.9 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
38.99617 5 0.0000 2.229 3 OF 12 ( 25.0%) 
NUMBER OF MISSING OBSERVATIONS = 54 


ee a th SOO oes em Gm rtm uh ime SUS cininy Sine Kt DGD UNO IED ey cna GOMES ANY SN STD Ane AED GD mn Gti GOPOD SA um inp) Sm SELLY GENE GWE GEE nl ct, Su OO Gm he Nh em um Gm clin GU GNP cit WO wt GD Sm GE Uwe GORE) Gem nem miki OUND Ge nim 


2.11 Job Area 


In examining the respondents' job areas, three categories are 
eliminated because of limited numbers of cases (Finance, 
Management, and Marketing) along with the Other category. 


The tables in Figures 2.lla and 2.11b represent the two questions 
with reportable results, although variables QUAL4 and QUALI11 are 
also reportable according to a chi-square Significance cutoff 
point of less than 0.02. However, the latter two tables have 5 
out of 12 cells (41.7%) with expected frequencies of less than 5, 
which would make any conclusions shaky at best. 


Fe 
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Figure 2.lla Read document from terminal 



























































QUALY 
COUNT 
ROW PCT Don't Have ROW 
Have TOTAL 
JOBAREA <-------- perenne teen + 
1 6 8 14 
Administrative 42.9 57.1 9.2 
+-------- +-------- + 
2 3 6 9 
Operations 3363 66.7 5.9 
+-------- +-------- + 
3 2 9 a1 
Customer serv 18.2 81.8 ye 
+—----~---- +-—------- + 
4 12 14 26 
Education 46.2 53.8 £7. 
+-------- +------~- + 
8 7 56 63 
Programming Lied 88.9 41.4 
+-—------- +—-~---—--- + 
9 7 22 29 
Research 24.1 75.9 19.1 
+—-------+ +--------— + 
COLUMN 37 115 152 
TOTAL 24.3 75.7 100.0 
CHI-SQUARE Disk. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
15.93324 5 0.0070 2.191 3 OF 12° ¢ 25.0%) 
NUMBER OF MISSING OBSERVATIONS = 77 


SS OR A Mite ml Set GD SE SOE GT Giae ee cm ome meee omen tee Seem GY Genes iG mS ae eee ee Gm en, Ge GED tetera et Stee Gm Wee me Gems nite wine Gee eS iy ems GoD ee me am me ams me ee ce 


Figure 2.lla shows minor differences that arise among respondents 
in various job categories with regard to the ability to read a 
document at the terminal. Programmers and researchers seem to 
have most contact with this source of information. 
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Figure 2.11b Talked to consultant or supervisor 
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COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
JOBAREA -------- toe em at at oe oe + 
A 7 ni 18 
Administrative 38.9 61.1 LOg7 
+—------ Cs eet + 
2 5 7 12 
Operations 41.7 58.3 Ty 
+——-----—- +-----—-- + 
3 7 3 10 
Customer serv 70.0 30.0 6.0 
+-—------- 0 teeteeeeeeteted + 
4 5 27 32 
Education 15.6 84.4 19.0 
+—-——----—--— +——------- + 
8 41 26 67 
Programming 61.2 38.8 39:39 
+-------- +—-------- + 
9 10 19 29 
Research 34.5 65.5 L753 
+-------- +—------- + 
COLUMN 72 93 168 
TOTAL 44.6 55.4 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
22.42744 5 0.0004 4.464 1 OF 12 ( 8.3%) 
NUMBER OF MISSING OBSERVATIONS = 61 


eo caer SO eam Me Me cate me Gee Ge Gee US ae SY GS Se ei aE ma eee een Cee an ae wien ete item meee ee ene cme tee Gm em mm ee me es nite ie eee re mY Oe OI SE GEO UT Ot GS in wine 


Figure 2.llb reveals a rather odd difference among respondents in 
various job areas. Other than those respondents in customer 
services, which is too small a category from which to draw 
conclusions, those in programming jobs say that they don't have 
access to consultants or supervisors as sources of information. 


2.12 Time Using Computers 


In examining the respondents’ time using computers, the lowest 
three categories are collapsed into one because of limited 
numbers of cases. 


Tables 2.12a and 2.12b represent the one question with reportable 
results, although all variations of question 10 are reportable. 
Tables producing a chi-square significance less than 0.02 are 
considered reportable. 
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Figure 2.12a Read printed document - Usefulness 









































USEFUL10 
COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
TIMECOMP SSS SS aS eae th a re tee ae 
3 22 Lek 33 
2 yrs or less 66.7 334-3 18.2 
; os eeeieeireteeieeteeteheals dectaeaetenteetente a 
4 16 20 36 
2 to 5 yrs 44,4 55.6 8 Be 
teen ee pe + 
5 21 34 35 
5 to 10 yrs 36.2 61.8 30.4 
+-------- +-------- + 
6 15 42 57 
10 or more yrs 26.3 73.7 35.5 
$-------- +-+------ + 
COLUMN 74 107 181 
TOTAL 40.9 Oo et 100.0 
CHI-SQUARE Dak. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
14.43662 3 0.0024 13.492 NONE 
NUMBER OF MISSING OBSERVATIONS = 48 


me ee ee ee wie ce ce crm i cee ee ee ce me ee ce ce ce ee eee tee ee te cee ee cee ee ee ee ce ee ree ee ee ee ee ee ce ee ee ee ee ee ee ee ee ee ee eee ee ee ee ee ee ee ee ee ee ee 


Figure 2.12a shows the first major interpretable results 
according to demographic information. Obviously, the more 
experience one has with computers, the more one relies on printed 
documentation as a source of information. The progression from 
33.3% of those with two years' or less experience with with 
computers uSing printed documentation to 73.7% of those with ten 
Or more years' experience is remarkable, particularly since the 
trend seems to smoothly increase as respondents increase in 
experience. 


Compare the table in Figure 2.12a with that in Figure 2.12b which 
shows that, overall, 82.7% of the respondents say they have 
access to printed documentation. However, while 95.2% of the 
experienced users say they have printed documentation, only half 
of those in the least experienced category claim that they do. 
One wonders whether there isn't any printed documentation, 
whether they are referring to printed documentation that exactly 
describes their particular tasks, or whether they just don't want 
to read and therefore deny its existence. For anyone who writes 
documentation, this is a discouraging result at best. 
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Figure 2.12b Read printed document - Quality 









































QUAL10 
COUNT 
ROW PCT Don't Have ROW 
Have TOTAL 
TIMECOMP £-------- 5 ies ae oe el ad oS aa a + 
3 16 16 32 
2 yrs or less 50.0 50.0 L6..3 
$-------- pone s-- + 
4 10 32 42 
2 to 5 yrs 23.8 76.2 21.4 
+—-------- +-------- + 
5 5 55 60 
5 to 10 yrs 8.3 oo ay 30.6 
+—-—-----—— +-------- + 
6 3 59 62 
10 or more yrs 4.8 953.2 31.6 
$—-------- t—------- + 
COLUMN 34 162 196 
TOTAL 17.3 82.7 100.0 
CHI-SQUARE DAF. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
35518052 2 0.0000 Soo. NONE 
NUMBER OF MISSING OBSERVATIONS = 33 


2.13 Interactive System Used 


Although the study did not intend to analyze the specific system, 
some relationships arise that could be interesting to those who 
are familiar with the systems represented in the study. In 
analyzing the specific systems, however, the CICS and SPIRES 
categories are removed since the numbers of respondents are too 
low for analysis. 
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Figure 2.13a System gives help on command 


















































USEFUL4 
COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
SYSTEM = --~------ ; Sanaa weed cal sb t==-S=-=— + 
2 10 30 40 
CMS 25.0 75.0 25.0 
+—-—-—----- +-------- + 
4 2 18 20 
SPEAKEASY 10.0 90.0 12.5 
$+-------- +—-------- + 
5 29 22 51 
SPF 56.9 43.1 31.9 
$-—------- +-------- + 
7 10 20 30 
TSO 33.3 66.7 18.8 
$-—------- $--—------ + 
9 8 pial 19 
WYLBUR 42.1 S749 be Pe 
$-------- $--—------ + 
COLUMN 59 101 160 
TOTAL 36.9 63.1 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
17.76706 4A 0.0014 7.006 NONE 
NUMBER OF MISSING OBSERVATIONS = “69 


The table in Figure 2.13a shows that the respondents tend to use 
the help facility for commands in SPEAKEASY, CMS, TSO, and to 
some degree with WYLBUR. However, respondents don't use that 
source of information with SPF. The latter two cases (WYLBUR and 
SPF) may result from unavailability at some installations (see 
Figure 2.13b.) 
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Figure 2.13b System gives help on command - by agreement 


















































QUAL4 
COUNT 
ROW PCT Don't Agreed ROW 
Agree TOTAL 
SYSTEM. - See eee fa SSeS toa SSS + 
2 5 35 40 
CMS 12.5 87.5 29%3 
+—-------- +—-~------- + 
4 2 19 21 
SPEAKEASY 9.5 90.5 13:33 
+-------- +-------- + 
ce) 14 38 52 
SPF 26.9 13e2 32.9 
t——------ +-------- + 
7 10 18 28 
TSO 35.7 64.3 by ae, 
$-—------- t-------- + 
9 jar | 10 17 
WYLBUR 41.2 58.8 10.8 
+-------- +—-—------- + 
COLUMN 38 120 158 
TOTAL 24.1 1959 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
10.39754 4 0.0342 4.089 1 OF 10 ( 10.0%) 
NUMBER OF MISSING OBSERVATIONS = 71 


It seems compelling then to turn to the quality question in an 
attempt to see whether respondents feel they have access to a 
help facility. Figure 2.13b shows the same basic pattern as the 
table in Figure 2.13a: most respondents know thay have such an 
information source available and most of them use it. Any 
attempt to evaluate the extent to which such a facility is liked 
or not is too fine an analysis for this sample size and produces 
unreliable results. 


Lé 
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2.14 Time Using System 


In examining the respondents' times using the specific system, 
the highest two categories are collapsed into one because of 
limited numbers of cases. 


Figure 2.14a System offered suggestions 
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Figure 2.14b Read printed document 






















































































QUAL10 
COUNT 
ROW PCT Don't Have ROW 
Have TOTAL 
TIMESYS 9 wre Looe s == +S + 
1 5 9 14 
< 1 month 357 64.3 one 
+------- +-------- + 
2 9 22 31 
1 to 6 mos 29.0 71.0 15.7 
+------- +-------- + 
3 17 57 74 
6 to 24 mos 23.0 77.0 37.6 
t—------- +-------- + 
4 4 52 56 
2 to 5 yrs are 92.9 28.4 
+—------- +-------- + 
5 22 22 
5 or more yrs 100.0 i242 
+------- +—-------- + 
COLUMN 35 162 197 
TOTAL 17.8 82.2 100.0 











EASY1 
COUNT 
ROW PCT Don't Use ROW 
Use TOTAL 
TIMBOYS- =“ -H==s=< ae seer as a + 
| 1 2 13 15 
< 1 month 13.3 86.7 cree 
$------- +-------- + 
2 8 26 34 
1 to 6 mos 23.5 76.5 16.2 
+------- +-------- + 
3 33 45 78 
6 to 24 mos 42.3 Siat S738 
+------- +-------- + 
4 | 32 28 60 
2 to 5 yrs 53.3 46.7 28.6 
ton +-------- + 
5 eS 12 23°." 
5 or more yrs 47.8 S252 oS Pe 
+-------- +-------- + 
COLUMN 86 124 210 
TOTAL 41.0 59.0 100.0 
CHI-SQUARE DF. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
13.31206 4 0.0098 6.143 NONE 
NUMBER OF MISSING OBSERVATIONS = 19 


The table in Figure 2.14a should be contrasted with the results 
shown in Figure 2.12a. In Figure 2.14a, respondents who have 
less experience with the particular system tend to use the 
systems' suggestions. However, as respondents increase in time 
on the system, they tend not to rely on this source of 
information. One wonders whether they are disillusioned or tend 
to rely more on themselves. Further analyses of respondents' 
likes and dislikes produces unreliable results. 


CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
16.23176 4 0.0027 2.487 2 OF 10 ( 20.0%) 


tl 
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N 


NUMBER OF MISSING OBSERVATIONS 


Compare the results shown in Figure 2.14b with those shown in 
2.12b. Again, the more time one spends on a system, the more one 
seems to at least acknowledge the existence of printed 
documentation as a source of information for problem resolution. 


8c 
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Figure 2.14c Talked to another user | 2.15 Experience on System 
Figure 2.15a Read printed document 
EASY a a a ee ee mee a a a a ee a a a ee re 
COUNT 
ROW PCT Don't Use ROW QUAL10 
Use TOTAL _ COUNT 
TIMESYS es reslaataantententantan sbeeteabeatententeneatant +-------- + ROW PCT Don't Have ROW 
. 1 5 10 15 . Have TOTAL 
< 1 month 33.3 66.7 7.0 EXPERIEN <---°----- tess ie eee r= + 
i ciara salto ee en ee + 1 16 31 47 
2 8 28 36 Not very 34.0 66.0° 23.9 
1 to 6 mos 22.2 77.8 16.9 | - toe $—------- + 
$o------- $-------- + 2 16 78 94 
3 23 59 82 Moderate 1730 83.0 | 47.7 
6 to 24 mos 28.0 72.0 38.5 | $-------- +—------- + 
a laa Ae aaa + 3 3 53 56 
4 32 25 57 Very Sod 94.6 28.4 
2 to 5 yrs 56.1 43.9 26.8 $-------- $o--- + 
teres cee ee + COLUMN 35 162 197 
5 9 | 14 23 | TOTAL 17.8 82.2 100.0 
5 or more yrs 3954 60.9 10.8 
Pa eat Seee ae ea aa + CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
COLUMN 77 136 213 14.46030 2 0.0007 8.350 NONE 
TOTAL 36.2 63.8 100.0 
NUMBER OF MISSING OBSERVATIONS = 32 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 . 
15.36550 4 0.0040 5.423 NONE 2 a en ee ee nr ere 
NUMBER OF MISSING OBSERVATIONS = 16 Rounding out the thoroughly consistent attitude respondents show 


. ; toward printed documentation, Figure 2.15a shows again that the 
a aa a a a a a a ae aa a more comfortable users feel with a system, the more they 


. acknowledge its manuals as a source of help. 
Finally, except for the strange anomaly of respondents who have 


used the system between two to five years (which is a significant 
period of time using a system), respondents seem to use other 
users aS a source of information (Figure 2.14c.) Some note 
should be taken of the general tendency to rely less on other 
users at the very beginning, relying on others in the first six 
months, and then trailing off as more time is spent on the 
system. 
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Figure 2.15b Read printed document - by agreement 
































QUAL10 
COUNT 
ROW PCT Don't Agreed ROW 
Agree TOTAL 
EXPERIEN £-------- === ap eae eee ee + 
| 1 6 25 31 
Not very 19.4 | 80.6 1901 
+------ +-------- + 
2 del 67 78 
Moderate ble ea 85.9 48.1 
+-------- +-------- + 
3 8 45 53 
Very ~~ Boye 84.9 S261 
+-------- +-------- + 
COLUMN 25 137 162 
TOTAL 15.4 84.6 100.0 
CHI-SQUARE Dak. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
0.47580 2 0.7883 4.784 1 OF 6 ( 16.7%) 
NUMBER OF MISSING OBSERVATIONS = 67 


As can be seen from the results shown in Figure 2.15b, once the 
respondents try written documentation regardless of how 
experienced they feel with the system, they are satisfied with 
its quality. 
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2.16 EXAMINING THE PROBLEM STATE 


To pursue the explanatory value of the user problem state, it is 
necessary (as noted in Section 2.8) to collapse the two "refresh" 
states (general and specific) developed by Dick Granda and 
Richard Halstead-Nusslock into a single category. 


Figure 2.16a Problem state vs. time using computers 


















































TIMECOMP 
COUNT 
ROW PCT |2 yrs or 2 to 5 5 to 10 10 or ROW 
less yrs yrs more TOTAL 
PSTATE <##es=<-= tose Se ata ate ers : aia aan tS + 
1 9 14 14 14 om 
Learn 17.6 27.5 27.5 7eF ie 23.0 
+-------- t—-------- +-------- $+ 
2 19 25 33 36 113 
Solve 16.8 22.1 292 31.9 50.9 
+-------- te----- +——------- tone + 
3 8 12 19 19 58 
Refresh 13.8 20.7 3258 32.6 26.1 
bs sleeleeteeteneaatinetenl +e ee eee $e ee ee ee : seeieeeieientnetenen + 
COLUMN 36 51 66 69 222 
TOTAL 16.2 23.0 29.7 3144 100.0 
CHI-SQUARE Dek, SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
1.48085 6 0.9608 8.270 NONE 
NUMBER OF MISSING OBSERVATIONS = 7 


The first step then is to try to determine whether the user 
problem state is an independent measure or whether it was really 
determined by level of experience. That is, the question is 
whether learning state is limited to beginners in computers or on 
the system and whether refresh state is limited to experienced 
users. Figure 2.16a shows that the problem state variable is 
independent from the variable that measures a respondent's years 
in computing. In other words, someone can be in a learning, 


problem solving, or refresh state nearly regardless of how long 


they've been in computing. 
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Figure 2.16b Problem state vs. time using system 
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TIMESYS 
COUNT 
ROW PCT under 1 to 6 6 to 24 2 to 5 5+ ROW 
1 month months months years years TOTAL 
PSTATE  -------- te +-—------- +o +—-------- +$—- + 
1 11 10 17 9 4 51 
Learn 21.6 19.6 3343 17.6 7.8 22.8 
ta rr ne tee ne ee teen ton $-------- + 
2 d 22 46 34 11 114 
Solve so 19:23 40.4 29.8 9.6 50.9 
+-------- +-------- +-------- +—-- +-------- + 
3 4 3 24 20 8 59 
Refresh 6.8 5.1 40.7 3369 13.6 26.3 
+—-~----—--— tame +—-------- $-- t—-- + 
COLUMN 16 35 87 63 23 224 
TOTAL aol tO 6 38.8 28.1 10.3 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
31.11296 8 0.0001 3.643 2 OF 15 ( 13.3%) 
NUMBER OF MISSING OBSERVATIONS = 5 


With regard to the amount of time a respondent spends on the 
specific system, one would expect that those in the learning 
state would be less experienced on the system. The table in 
Figure 2.16b shows that over 40% of those respondents in the 
learning state have been on the system six months or less, and 
nearly three-quarters for two years or less. Similarly, from 
Figure 2.l16c, the same nearly 40% of those in the learning state 
categorize themselves as not very experienced with the system. 


Respondents in the problem solving state shouldn't necessarily be 
more or less experienced since problems occur at all levels. 

Both tables show that the bulk of the respondents in the problem 
solving state fall in the middle categories, 40.4% having from 
six months' to two years’ time on the system and 50% evaluating 
themselves as moderately experienced. 


Chapter 2: Results 33 


Figure 2.16c Problem state by experience on system 









































EXPERIEN 
COUNT 
ROW PCT |Not very Moderate Very ROW 
TOTAL 
PSTATE =o wt rer +—-—------- $-—-- $——------- + 
1 21 20 10 51 
Learn 41.2 39.2 19.6 22.8 
tenn t—-------- t-------- + 
2 24 57 33 114 
Solve 21.1 50.0 28.9 50.9 
+——------ teen S enateteatentntaten + 
3 13 26 20 59 
Refresh 22.0 44,1 33.9 26.3 
+—-------- +—-—------- +-~------- + 
COLUMN 58 103 63 224 
TOTAL 25.9 46.0 28.1 100.0 
CHI-SQUARE Dakss SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
8.96879 4 0.0619 13.205 NONE 


NUMBER OF MISSING OBSERVATIONS = 5 
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Finally, one might expect respondents in the refresh state to 
tend to be more experienced. The table in Figure 2.16b shows 
that 47.5% of the respondents in refresh state have had at least 
two years’ time on the system and Figure 2.16c shows that 78% of 
them say that they are at least moderately experienced. 


Therefore, the user problem states are independent of the number 
of years one has worked in computing and are related to, but not 
determined by, the amount of experience one has with the specific 
system. The difference between user problem states and 
experience on a system is that authors of interactive systems may 
not be able to design a system that can react to a user problem 
based on some measure of experience but might be able to put 
hooks into a system that can react to the user's problem state. 


2.17 Was the Source Used? 


The next step in analysis of the user problem state is to 
crosstabulate it with the 48 opinion questions based on whether 
the information source is used or not. Again, use is determined 
by the respondent answering any of the four agree-disagree 
choices. Overall quality of the source did not produce any 
reportable results, so tables are shown only for any of the other 
three attributes that did produce reportable results. 


Tables producing a chi-square significance less than 0.05 are 
considered reportable for this stage of the analysis since a 
criterion of 0.02 is too restrictive. This means, however, that 
the chances of making erroneous conclusions from the tables 
increase. 
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You can assume that any question for which a table is not shown Figure 2.17b System provides detail on message 
does not reveal any differences among respondents in the three 2 ttt nt er ee ee 
problem state categories. For example, question 10 on printed 
documentation shows that more respondents use manuals as a source EASY2 
of information than don't use them, but doesn't show any COUNT 
differences when crosstabulated with the three problem state ROW PCT Don't Use ROW 
categories. use TOTAL 
PSTATE De a mie en ai Se 5 dhe eia alan + 
Figure 2.17a System offered suggestions 1 13 33 46 
SS tt eR SR Re ee mn am ee mim Learn 26.3 Tied 22.5 
+-------- +—------—- + 
EASY1 2 58 46 104 
COUNT Solve 55.8 44.2 51.0 
ROW PCT Don't Use ROW sani a cals fsSoSSS>= + 
ou use TOTAL 3 31 23 54 
PSTATE ~------- $-------- +-------- + Refresh 57.4 42.6 26.5 
1 12 34 46 a acpi ania iatar aa a + 
Learn 26.1 73.9 22.4 COLUMN 102 102 204 
$-------- $-------- + TOTAL 50.0 50.0 100.0 
2 44 61 105 
Solve 41.9 58.1 51 w2 CHI-SQUARE DF SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
$o-—----- $oeee + 11.26545 2 0.0036 23.000 NONE 
3 27 27 54 . 
Refresh 50.0 50.0 26.3 NUMBER OF MISSING OBSERVATIONS = 25 
$e eee $e + 
COLUMN 83 122 COB a i ee 
TOTAL 40.5 59.5 100.0 
Figure 2.17b shows a more pronounced version of the same trend. 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 Respondents in a learning state clearly use a system's ability to 
6.07447 2 0.0480 18.624 NONE provide detail on a message, whereas respondents in the other two 
| categories don't. 
NUMBER OF MISSING OBSERVATIONS = 24 
Figure 2.17c System outlines current situation 
The results in Figure 2.17a show the beginning of a trend. EASY3 
Respondents in the learning state tend to use a system's COUNT 
Suggestions, whereas those in a problem state use this source ROW PCT Don't Use ROW 
less, and those ina refresh state use it even less. use TOTAL 
PSTATEB 09 wero SSS S tess <se9< + 
1 19 24 43 
Learn 44.2 55.8 Ze 
+-------- +-------- + 
2 62 41] 103 
Solve 60.2 39.8 51.68 
t------- - ¢-- - - - + 
3 37 16 53 
Refresh 69.8 30.2 26.6 
+-------- : seatetententesteatenten + 
COLUMN 118 81 199 
TOTAL 59.3 40.7 100.0 


CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
6.53006 2 0.0382 Lis5G3 NONE 


NUMBER OF MISSING OBSERVATIONS = 30 


Go 
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Although respondents, regardless of their problem states, do not. Figure 2.17e Talked to consultant or supervisor 
seem to use a system's ability to outline the current Situation 4 2 2 2 4 2 4 4 tt re rr nnn 
(Figure 2.17c), those in the learning state again tend to buck 






























































the trend with a slight majority of them saying that they used USEFUL12 
this source of information. COUNT 
ROW PCT Don't Use ROW 
Figure 2.17d System gives help on command use TOTAL 
See ar Se el ete i na Rr ae tage ae en i a i ree PSTATE mf al co 
af 12 28 40 
EASY4 Learn 30.0 70.0 2204 
COUNT , iaetenantenieataatend , aaheiemetetenis + 
ROW PCT Don't Use ROW 2 38 54 92 
use TOTAL Solve 41.3 58.7 50.8 
PSTATE -<3------~ +-------- +-------- + +~-------- +—-------- + 
i 8 40 48 3 33 16 49 
Learn 16.7 83.3 23.2 Refresh 67.3 3257 Zrel 
+-------- +-------- + +-------- t—-—--- + 
2 4] 64 105 COLUMN 83 98 181 
Solve 39.0 61.0 50.7 TOTAL 45.9 54.1 100.0 
+-------- +—--—------ + 
3 19 35 54 CHI-SQUARE Dek. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
Refresh S52 64.8 26.1 13.93316 2 0.0009 18.343 NONE 
+-------- +-------- + 
COLUMN 68 139 207 NUMBER OF MISSING OBSERVATIONS = 48 
TOTAL 32.9 67.1 100.0 
CHI~SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
7.66077 2 0.0217 15:3:766 NONE Finally, the greatest differences among the three problem state 
categories shows up in Figure 2.17e. While 70% of the respondents 
NUMBER OF MISSING OBSERVATIONS = 22 in the learning state say they talk to a consultant or another 
user, less than one-third of those in a refresh state say they 
SS SSS aS SS a ee a eS eae Se eee eS See ee eee do. 
The table in Figure 2.17d continues the trend, this time One might be tempted at this point to conclude that the only 
regarding help messages provided on command. Although all difference among refresh, solving, and learning states is the 
categories use this source of information, learners make notably degree to which a learner will try everything while someone with 
greater use. a refresh problem tends to rely more on memory (or some other 


source not named in the survey or simply perseveres.) However, 
because one tries a source says nothing much about the success of 
that source in helping resolve the user's dilemma. 


2.18 How Did They Evaluate the Source? 


For the next stage of anaysis, the obvious follow-up to analyzing 
whether a respondent uses a source is to analyze whether they 
like it. To eliminate small-cell problems, both agree choices 
are lumped together and so are both disagree choices (agreement 
means the degree to which the respondent agrees with the 
Statement to the effect that the particular source of information 
is easy to use, understandable, easy to access, etc.; see 
Appendix A.) Respondents indicating they do not use the source 
are eliminated along with those who did not answer the question. 


Tables producing a chi-square significance less than 0.05 are 
again considered reportable. 
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Figure 2.18a System offered suggestions 
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EASY1 
COUNT 
ROW PCT 
PSTATE ee a tasatsstee es 
1 20 
Learn 58.8 
fea lis Seen 
2 36 
Solve 59.0 
tcneones 
3 6 
Refresh 22.2 
lane elec 
COLUMN 62 
TOTAL 50.8 


Disagree Agree 











pew ee ee ee 

14 

41.2 

fe ae te eee eee 

25 

41.0 

pew e enn 

21 

77.8 

7 ren 
60. 
49.2 


CHI-SQUARE D.F. SIGNIFICANCE 
11.34602 2 0.0034 


The table in Figure 2.18a shows the beginning of a second trend 
with regard to user problem states and the various sources of 
information. Respondents in the learning state may use a 
system's suggestions more than the others (see Figure 2.17a), but 
they like it a whole lot less. 
refresh state don't use this source of information much, but they 
like it when they do. Respondents in the problem solving state 
tend to react more like learners with regard to system 


suggestions. 











122 
100.0 


MIN E.F. 
13.279 


On the contrary, those in the 


CELLS WITH E.F.< 5 
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Figure 2.18b System provides detail on message 
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UNDER2 
COUNT 
ROW PCT 
PSTATE <-------- Laeecomne 
Z | 15 
Learn 53.6 
+e ee eee 
2 29 
Solve 72.5 
ae a eh Sees 
3 5 
Refresh 26.3 
fe we oe ee wee eee ee we 


COLUMN 49 
TOTAL 56.3 


Disagree Agree 











he a es ees 

13 

46.4 

fe am nin ote eo 

11 

27.5 

ET 

14 

73.7 

nn 
38 
43.7 


CHI-SQUARE D.F. SIGNIFICANCE 
11529578 2 0.0035 


NUMBER OF MISSING OBSERVATIONS = 











87 
100.0 


MIN E.F. 


8.299 


CELLS WITH E.F.< 5 
NONE 


The results in Figure 2.18b again show that respondents in a 
learning state don't particularly like the results of message 
Getails, certainly not as much as respondents in a refresh state, 
even though learners use this source of information more (Figure 
2.17b.) Respondents in the problem solving state are remarkably 
unlike the other two categories with regard to system message 
details in that they seem to be much more disappointed in the 


results. 
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Figure 2.18c System gives help on command 
































UNDER4 
COUNT 
ROW PCT {Disagree Agree ROW 
TOTAL 
PSTATE 8 wer----- $e eee teen ene + 
1 15 21 36 
Learn 41.7 58.3 29.3 
+-------- +-------- + 
2 27 31 58 
Solve 46.6 53.4 47.2 
+—-------- +-------- + 
3 3 26 29 
Refresh LOGS 89.7 23.6 
+-------- $-------- + 
COLUMN 45 78 123 
TOTAL 36.6 63.4 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
11.49070 2 ~ 0.0032 10.610 NONE 
NUMBER OF MISSING OBSERVATIONS = 106 
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The trend continues with help messages on command. From the 
table in Figure 2.18c, respondents in a refresh state are pleased 
with results, whereas those in the learning state are notable 
less pleased. Problem solvers are nearly evenly split. 


Figure 2.18d System gives example 
































USEFUL6 
COUNT 
ROW PCT |Disagree Agree ROW 
. TOTAL 
PSTATE = te +e tome + 
1 10 2 12 
Learn 83.3 16.7 24.5 
+-------- +-------- + 
2 14 11 25 
Solve 56.0 44.0 51.0 
+-------- +-------- + 
3 4 8 12 
Refresh 33.3 66.7 24.5 
- Perse rere eee + 
COLUMN 28 21 49 
TOTAL 57.1 42.9 100.0 


CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
6.15222 2 0.0461 5.143 NONE 


NUMBER OF MISSING OBSERVATIONS = 180 
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Figure 2.18d is shown more for the sake of completeness than for 


interpretable results. Whereas the learning vs. refresh states 
seem to show the same dislike-like trend with regard to online 
examples, the small numbers in both categories tell us to make 
conclusions cautiously. 


Figure 2.18e Tutorial at terminal 
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UNDER7 
COUNT 
ROW PCT |Disagree Agree ROW 
TOTAL 
PSTATE = 3-------- eS Sa ee $e + 
Z 14 7 21 
Learn 66.7 i BS A 26.9 
+--—------ +-------- + 
2 13 24 37 
Solve 35.1 64.9 47.4 
+-------- +-------- + 
3 4 16 20 
Refresh 20.0 80.0 25.6 
+-------- +-------- + 
COLUMN 31 47 78 
TOTAL 39.7 60.3 100.0 
CHI-SQUARE DF. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
9.93979 2 0.0069 7.949 NONE 
NUMBER OF MISSING OBSERVATIONS = 151 


Likewise, the results in Figure 2.18e show the same trend, but 
are based on only 78 out of 229 possible respondents, reflecting 
the general unavailability or lack of use of tutorials as a 
source of information. 


ae 
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Figure 2.18f Read printed document 
































USEFUL10 
COUNT 
ROW PCT |Disagree Agree ROW 
TOTAL 
PSTATE -------- tosses aia + 
HI 4 21 25 
Learn 16.0 84.0 23.8 
+—-------- +--------+ 
2 20 35 50 
Solve 36.4 63.6 52.4 
+-------- +-------- + 
3 2 23 25 
Refresh 8.0 92.0 23.8 
$-------- $-------- + 
COLUMN 26 79 105 
TOTAL 24.8 75.2 100.0 
CHI-SQUARE D.F. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
8.77402 2 0.0124 6.190 NONE 
NUMBER OF MISSING OBSERVATIONS = 124 


Documentation, which shows up as somewhat of a recognized source 
of information among more experienced respondents (Figures 2.12a, 
2.12b, 2.14b, and 2.15a), shows up in Figure 2.18f as a favorite 
of respondents in a refresh state. However, note the shift among 
learners from uSing a source and disliking it to using it and 
liking it. It seems that printed documentation is everyone's 
favorite source of information, with problem solvers lagging 
somewhat behind the bandwagon. 
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Figure 2.18g Talked to another user 
































QUALI11 
COUNT 
ROW PCT {Disagree Agree ROW 
TOTAL 
‘PSTATE <-3-3-"-~ SS SSS =sSS=<= + 
1 7 32 39 
Learn 179 82.1 22.2 
+-------- +-------- + 
2 4 86 90 
Solve 4.4 95.6 Sid 
$—-----+--+-------- + 
3 8 39 47 
Refresh 17.0 83.0 26.7 
+-------- +-------- + 
COLUMN 19 L157 176 
TOTAL 10.8 89.2 100.0 
CHI-SQUARE Dik. SIGNIFICANCE MIN E.F. CELLS WITH E.F.< 5 
7.73365 2 0.0209 4.210 1 OF 6 ( 16.7%) 
NUMBER OF MISSING OBSERVATIONS = 53 


Talking to another user 1S very popular among all respondents. 
Figure 2.18g shows learners and refreshers pleased with this 
source and also shows problem solvers leading the parade this 
time with a remarkable 86 to 4 majority. This the the only 
reportable result generated by crosstabulating the problem states 
with how the respondent judged the overall quality of the 
information source rather than some attribute specific to the 
problem being resolved. 


2.19 Conclusions about Problem States 


From the results of this study, conclusions can be drawn 
regarding users of interactive systems and their possible 
behaviour patterns in response to various sources of information 
when confronted with a problem. 


e A user's problem state, if it can be identified, can be useful 
in determining which sources of information are used and which 
are successful in helping resolve the problem. 


e A user's problem state is somewhat related to, but not fully 
determined by, the amount of experience the individual has had 
with the particular system. It is however entirely independent 
of the amount of time the user has been in computing. 


e Users in a learning state differ from those in other states in 
that they tend to try many sources of information but find help 
only with printed documentation (and system help messages to a 
lesser degree.) 
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e Users in a problem solving state don't differ greatly from the 
other states, sometimes acting more like those in a learning 
State and sometimes like those in a refresh state with no 
notable pattern. 


e Users ina refresh state differ from those in either of the 
other states in that they tend to use information sources 
listed in the survey less, but they are more satisfied when 
they do use them. 


e The lack of use and particularly the lack of success of many 
sources of information is a signal to software authors that 
more thought and work is required. 


2.20 FACTORING THE QUESTIONS 


One conclusion drawn from analyzing the survey is that it is too 
complicated. The fact that several of the questions like "help 
on subject area," "System gives example," and "listed resource 
index" seldom or never produce interpretable results points to 
the possibility that there is a smaller classification subset of 
sources of information that corresponds more closely with the way 
users understood their interactions with computers. Likewise, 
the fact that the four ease of use, understandability, 
accessibility, and even overall quality attributes nearly always 
produce the same results leads one to explore the possibility 
that these are not really different measures but different 
dimensions of a smaller number of measures. Factor analysis is 
therefore a logical next step. 


In all factor analyses reported below, Principal Components 
extraction, Varimax rotation, and listwise deletion of cases with 
missing values were used. The appropriate number of factors was 
theoretically determined rather than using an eigenvalue, scree, 
or percent-variance-explained criterion. 


Only the rotated factor loading matrix is shown. Loadings below 
0.6 are blanked out and variables are sorted according to their 
loadings in factor order to assist in interpretation. 


2.21 Examining the Like--Dislike Dichotomy 


Four factor analyses are reported on the four attribute sets of 
questions using the dislike/like dichotomy. These analyses 
should provide a hint whether respondents tend in their own minds 
to group the twelve sources of information according to whether 
the source is liked or not. 


The factor analyses limit the number of factors to 2 basen on the 
theory that there are really two underlying dimensions: machine 
and nonmachine sources of information. 

In the following four sets of analyses, the resulting eigenvalues 
are always characterized by a large value for factor 1 and a much 
smaller one for factor 2, the second one always below 1.0. This 
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means that there is only one factor in each analysis. Whatever 
variables show up on the second factor are really just another 
dimension or residual of the first. However, how the variables 
fall out on the two dimensions and whether they are substantively 
interpretable is of interest. 


Figure 2.2la Accessibility 
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FACTOR 1 FACTOR 2 
EASY2 -85984 
EASY6 -83679 
EASY11 -82332 
EASY4 -80820 
EASY3 - 76330 
EASY1 -69652 
EASY9 -60511 
EASY7 ~82171 
EASY12 80767 
EASY5 -76761 
EASY8 ~69052 
EASY10 -62616 


The rotated factor matrix shown in Figure 2.2la shows two 
dimensions of accessibility that are not easy to interpret. That 
is, it is hard to interpret the grouping on factor 1 based on use 
of online sources (questions 1 through 4, 6, and 9) and on 
talking to another user. Variables loading on factor 2 are 
equally uninterpretable. 


Figure 2.2l1b Understandability 


FACTOR 1 FACTOR 2 
UNDER4 -87223 
UNDER6 -84297 
UNDER2 -81334 
UNDER11 ~ 76495 
UNDER] ~71459 
UNDER9Y -68388 
UNDER1 2 -65885 
UNDER7 . 86808 
UNDER5 . 78657 
UNDER1 0 71985 
UNDER8 -64602 
UNDER 3 -62700 -64056 


The results shown in Figure 2.21b are equally puzzling. In 
factoring the understandability question, this time several 
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online sources of information load on the same factor as talking 
both to another user and to a consultant or supervisor. In fact, 
glancing ahead to Figures 2.2lc and 2.21d, the same lack of 
pattern and interpretability characterize the rotated matrices 
for usefulness and quality. Thus, it doesn't seem informative to 
factor the questions based on whether the respondents likes the 
various sources of information or not. 


Figure 2.2lc Usefulness 


FACTOR 1 FACTOR 2 
USEFUL1 -88608 
USEFUL2 -88448 
USEFUL3 .85796 
USEFUL8 82943 
USEFUL9 .82769 
USEFUL5 75208 
USEFUL6 -65254 -60738 
USEFUL4 -64724 63639 
USEFUL10 -64040 -61503 
USEFUL12 88824 
USEFUL11 sf 7912 


USEFUL7 
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FACTOR 1 FACTOR 2 
QUAL2 .91546 
QUAL6 91462 
QUAL3 84623 
QUAL1 84254 
QUAL8 .76120 
QUALS .75552 
QUAL4 64832 63394 
QUAL12 .64773 61946 
QUAL11 .60372 
QUAL10 .95570 
QUALS .72294 
QUAL? 63677 64391 


ee ets ce ee em te em ieee eee cee eee ee ee mm me ce em Sere cm eer omen mem ne ee ee ee eee me ree wee Sem eee ete ee ee i me eee ee ee ee eee ee ee ee ee ee en es ee eee ee oe ae ee me me oe 


2.22 Examining the Use--Not Use Dichotomy 


Again four factor analyses involve the four attribute sets of 
questions, this time using the use/not use dichotomy. These 
analyses should provide a hint whether respondents tend to group 
the twelve sources of information according to whether the source 
is used or not. 
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The factor analyses are again limited to 2 factors still looking 
for what theoretically are the two underlying dimensions, machine 
and nonmachine sources of information. 


Figure 2.22a Accessibility 


FACTOR 1 FACTOR 2 
EASY3 .84526 
EASY2 .83808 
EASY6 .80360 
FASY1 .79257 
EASY9 e74771 
EASY5 .74051 
EASY4 .70619 
EASY7 .65950 
EASY8 63382 
EASY12 .84832 
EASY11 .81557 
EASY10 .68504 


The results of the factoring of the accessibility questions are 
shown in Figure 2.22a. Results are more satisfying in that all 
online sources of help factor in the first dimension and all 
nonmachine sources in the second. 


Figure 2.22b Understandability 


FACTOR 1 FACTOR 2 
UNDER6 .83808 
UNDER5 ~82103 
UNDER7 «82013 
UNDER 2 -81120 
UNDER8 ~79412 
UNDER1 . 78004 
UNDER3 ~77543 
UNDERSY ~ 76297 
UNDER4 -6854] 
UNDER11 -84043 
UNDER1 2 <7 9098 
UNDER10 ~ 74234 
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In fact, the remaining factor matrices in Figures 2.22b through 
2.22d show the same separation for understandability, usefulness, 
and overall quality. While this result may seem obvious to a 
certain degree, it seems likely that, although these nonmachine 
sources of information had to be included in the study, they may 
somehow cloud respondents' reactions to the machine sources. At 
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least in further studies of online sources, these "offline" 


sources should be segregated. 


Figure 2.22c Usefulness 


FACTOR 1 
USEFUL2 86084 
USEFUL1 84456 
USEFUL8 81247 
USEFUL9 78616 
USEFUL3 .78141 
USEFUL5 .77168 
USEFUL4 74964 
USEFUL6 . 73677 
USEFUL7 .71315 
USEFUL12 
USEFUL11 
USEFUL10 -60364 
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FACTOR 1 
QUAL2 «89272 
QUAL3 -85283 
QUAL1 -84924 
QUAL4 81330 
QUAL6 -80822 
QUAL5 - 76132 
QUAL7 #73503 
QUAL8 -68621 
QUALY — .64494 
QUAL10 
QUAL12 
QUAL11 
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FACTOR 2 


86974 
. 78697 
66879 


FACTOR 2 


83004 
81320 
75256 


2.23 Trying a Two--Attribute FACTOR 


AS an experiment to begin testing the independence of the 
accessibility, understandability, usefulness, and quality 
attributes, two factor analyses (based on the not used/used 
dichotomy) including twelve sources from each of two attributes 


are reported. 
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Figure 2.23a Understandibility and accessibility 
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FACTOR 1 FACTOR 2 
UNDER6 85979 
UNDERY 85818 
UNDER8 ~85203 
UNDER4 «83975 
UNDER5 «83133 
UNDER7 ~82426 
UNDER3 ~82412 
UNDER12 ~82349 
UNDER2 81309 
UNDER1 79262 
UNDER10 - 78673 
UNDER11 . 78584 
EASY5 .85797 
EASY3 - 83646 
EASY2 -82635 
EASY6 ~82446 
EASY1 -81662 
EASY9 -80337 
EASY8 78165 
EASY7 73817 
EASY12 s69193 
EASY11 ~68177 
EASY10 - 66458 
EASY4 - 64010 


Figure 2.23a shows the results of factoring accessibility and 
understandability together where the evidence is strong that they 
are separate measures, at least from each other. 


Since quality is supposed to be a separate dimension from the 
other three attributes, the second factor analysis is shown for 
accessibility and quality. The results are the same: the two 
attributes produce two distinct factors. 


The eigenvalues produced by these two two-attribute analyses are 
different. The accessibility-understandibility pair produced 
three eigenvalues greater than 1 (15.2, 2.7, and 1.1). 

Therefore, the analysis was rerun using the eigenvalue-1l 
criterion which produced three factors. The first factor is 
still all of the usefulness questions. The second and third 
factors are the accessibility questions separated with the 
machine sources loading highest on factor 2 and the nonmachine 
sources on factor 3. The accessibility-quality pair produces two 
eigenvalues greater than 1, so the analysis was not rerun. 


GE 
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Figure 2.23b Understandibility and quality 


Figure 2.24 Four dimensions into three 
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2.24 Comparing the Four Attributes 


The final step in this second round of analysis of the SHARE 
quesionnaire data was to factor all four attributes to explore 
indications that maybe there are fewer than four real measures. 
The results of the analysis are shown in Figure 2.24. Although 
four factors were requested, only three interpretable factors 
result. In fact, the scree rule also indicates three factors and 
the percent variance explained by those three factors is nearly 
75%. 


Clearly, the usefulness and understandability attributes are 
candidates for being combined into a single dimension. The 
overall quality attribute seems to hold its own along with 
accesSibility. Therefore, it would seem productive to condense 
the number of questions asked in a survey of this type to at 
least three: how easy is it to get to the source of information 
to resolve a specific problem, how successful is it in helping 
solve that specific problem, and how good a source of information 
is it compared overall with other sources? 
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COORDINATOR PACKET 


June 15, 1981 


Dear Recruitee: 


Enclosed please find the definitive SHARE Human Factors Project 
Questionnaire packet, consisting of: 


1. "System Tailoring Information" questionnaire for coordina- 
tors 
2. "Obtaining Assistance While Using Computers Interactively" 


questionnaire for users (in case you've misplaced your orig- 
inal copy) 


s Cover letter explaining more about these Human Factors Pro- 
ject questionnaires. 


We apologize for the delay in mailing this material to you; how- 
ever, distribution of the first and last items was approved only 
recently' by the SHARE Board and Advisory Council in May at the 
Interim in Salt Lake City. 


So far, we have not received nearly the thousand responses we 
need for statistically significant analysis. If you can possibly 
obtain more completed user questionnaires, please do so. For 


those of you who have already returned some user questionnaires, 


please also now complete and return the "System Tailoring Inform- 


ation” questionnaire. 


We will discuss the status of this activity in Chicago at Session 
A056 on Wednesday, August 26 at 4:30 p.m. Hope to see you there! 


Joan M. Winters (SLA) 
Deputy Manager, Human Factors Project 


Seana O'Hare (SLA) 


Stara, Olen. 
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May 8, 1981 


Dear Questionnaire Coordinator: 


SHARE is an organization of users of large-scale IBM systems. Pt 2s 
divided into working projects, each of which has a common interest 
relating to IBM software products and application areas. The 
various SHARE projects maintain dialogues with IBM concerning future 
product developments. It is crucial for these dialogues that 
projects keep abreast of the ways in which computer systems, 
including non-IBM systems, are being used in customer installations 
and the problems that individual users are encountering with current 
systems. 


Since early in 1980, a committee of the SHARE Human Factors Project 
Cin cooperation with the Documentation and Standards Project) has 
been working to develop an anonymous questionnaire evaluating 
interactive user assistance needs. We hope to be able to use the 
results of the survey to understand better the informational needs. 
of interactive users and to aid in the dialogue with IBM. 


To obtain statistically significant information, we need at least 
one thousand responses from a very wide variety of end-users using 
various systems. In order to get this kind of response, we decided 
to distribute the questionnaires to volunteer coordinators who will 
redistribute them to end-users at their installations. 


In addition to the end-user questionnaire, a second questionnaire 
was developed to be filled out by you, the installation coordinator, 
so that in the statistical analysis we can take into account 
installation tailoring of vendor-provided online documentation. 


We expect that you will take whatever actions are necessary before 
distribution at your installation to explain the purpose of the 
questionnaire, e.g., a cover letter and/or a meeting. Also, if you 
only run one interactive system with online documentation, you may 
be able to fill in the response to Question 5 before duplicating the 
end-user questionnaire for distribution. Note that the response to 
Question 5 should identify the most "user-visible”™ major component 
of the system that provides online documentation to the user (e.g., 
SPEAKEASY, not CMS, if the user is running SPEAKEASY under CMS). 


To return the responses to us, please collect all of the end-user 
questionnaires at your installation, add acompleted copy of the 
System Tailoring Information questionnaire and send to: 


Seana O'Hare 

P.O. Box 4349 

SLAC Bin 97 
Stanford, CA 94305 


TF 
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We will analyze the responses, preserving installation anonymity, 
and present preliminary results at SHARE 57 in Chicago this August. 
We will also send the final results to any installation 
participating in this survey. 


We thank you for any cooperation you can provide. 


Joan M. Winters (SLA) 
Deputy Manager, Human Factors Project 


Seana O'Hare (SLA) 


Seana O-line 
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System Tailoring Information Questionnaire 
Interactive User Assistance 


Most installations have tailored their interactive user assistance. 
In order to understand the data we obtain from end-users at your 
installation, we need some information from you. as questionnaire 
coordinator about any changes you have made in this area. Please 
answer these questions in as much detail as you think necessary so 
that we can understand your online documentation environment. 
Without infringing on installation anonymity, we will correlate this 


questionnaire with the end-user questionnaires from your 
installation. If it would help us understand more about your system 
than what we have asked you for, please feel free to explain, and 


attach additional sheets if necessary. 


Please fill out one of these questionnaires for each system you run 
that appears in the end-user questionnaire responses, oe re the 
response to Question 5. 


(1) Have you added HELP files to your system? 


(2) Have you modified the format of the text presented by HELP? 


(3) Have you modified the vendor's HELP processor? 


(4) Have you written your own HELP processor? 








(5) Have you made other additions or modifications to HELP? 








(6) Have you added other online user assistance facilities? Please 
describe briefly. 


oF 
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(7) Have you done anything to make it easier for people to find the 
information they need online? 


(8) Is there anything else about your system that we should know in 
order to understand your data (e.g., do you call a system a name 
other than its standard name)? 


June 1981 


Appendix A: Packet 57 
A Questionnaire on 
Obtaining Assistance while Using Computers Interactively 
Sponsored by the SHARE Human Factors Project 


in Cooperation with the 
Documentation and Standards Project 


We would appreciate your cooperation in filling out the following anonymous questionnaire. 
Its purpose is to get your opinion about the quality of different types of information 
that you might have tried to use during an interactive computing session. The first part 
of this questionnaire asks you for background information. The main part asks you to 
recall and evaluate a specific informational situation. The last part requests a general 
appraisal of the information resources available on your system. 

We will use the results to help us to understand better the informational needs of 
interactive users and to formulate requirements for submission to IBM. The results of 
this survey will be published in the Proceedings of SHARE. 


A. BACKGROUND 
First tell us a little bit about yourself: 


1) What is the primary business of your company or institution? 
(1) Business Services: Hotels, Amusement, or Non-profit Organizations 


(2) Computer and Data Processing, including Software Services, Service Bureaus, 
Time Sharing, and Consulting 


(3) Construction: Mining and Agriculture 

(4) Education: Colleges, Universities, and other Educational Institutions 
(5) Finance: Banking, Insurance, Real Estate, Securities, and Credit 

(6) Government: Federal, State, and Municipal, including Military 

(7) Manufacturing 

(8) Medical and Legal Services 

(9) Trade: Wholesale and Retail 
€10) Transportation Services: Land, Sea, and Air 

C11) Utilities: Communications, Electric, Gas, and Sanitary Services 


C12) Other 


2) To which area is your job most related: 
_...._—s«—d (‘1s Administrative services 

(2) Computer operations 

(3) Customer services 

€4) Education 

(5) Finance 

(6) Management 

(7) Marketing 

(8) Programming 


(9) Research 


C10) Other 
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3) How long have you been using computers? 

(12) +1 month or less 
C2) More than 1! month but not more than 6 months 
(3) More than 6 months but not more than 2 years 
(4) More than 2 years but not more than 5 years 
(5) More than 5 years but not more than 10 years 


(6) More than 10 years 
B. SPECIFIC SITUATION 


In order to give us some insight into the types of informational needs of interactive 
users, we would like you to recall a specific situation within the last month when you 


were working at a computer terminal and you needed information. You may have needed 
information to solve a problem, to try something new, to correct an error, or for some 
other reason. Please answer the following questions as they relate to this situation and 


your information needs: 


4) Briefly describe the situation. Include the type of information you were seeking. 
List the different sources you used, and tell us which one was successful. 


cP 


5) What interactive system were you using in the situation you just described? 


a. (1) CICcs cee, REY SPF ___ (9) WYLBUR 

___ (2) CMS _...-_—»«-—d«nA“<« 6) ~SPIRES __. (10) Other (please specify) 
—_. -(3) MTS __.. +«s«( 7) «~+TSO 

—__.. (4) -SPEAKEASY iu, 66) TSS —___. «C119 Don't know 


6) How long have you been using this system? 


C1) 1! month or less 


(2) More than 1! month but not more than 6 months 


(3) More than 6 months but not more than 2 years 
(4) More than 2 years but not more than 5 years 
(5) More than 5 years but not more than 10 years 


(6) More than 10 years 


7) How experienced a user do you consider yourself on this system? 


(1) Not very experienced 


(2) Moderately experienced 


(3) Very experienced 
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The following three questions ask you to evaluate the accessibility, understandability, 
and usefulness of the information sources you used. 


Note that all systems do not have all types of information resources listed belon. 


8) For. those information resources you used in the situation you described above, rate the 
following on the scale provided: 


| The information I tried to | | | | | | 
| obtain was easy to get to Strongly | Strongly || Didn't | 


When Io... | agree Agree Disagree | disagree use 










(1) Was offered suggestions by | 
the system about what to i 
do next { | 


eet 
eed 
ee ees come 
ae 
ane eae ste 


| (€2) Asked the system for more { | 
detail about a message | | | 


(3) Asked the system about my | | 
| current situation and | | 
possible options | | | 


(4) Asked the system for help 
on a command(s) 


(5) Asked the system for help 
about a subject area (e.g., 
FORTRAN or Inventory) 


(6) Asked the system for an 
example 





(7) Used a tutorial at the | 
terminal | 


| ¢€8) Listed an index of | | 
available information | 
resources at the terminal 


| (€9) Read (part of) a document | i i | { 
on the terminal | a 


| €10) Read (part of) a printed | | | | 
| copy of a document | 


(11) Talked to another user | 


| (12) Talked to a consultant or | | | | 
supervisor 


} €13) Other (describe) 
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9) For those information resources you used in the situation you described above, rate the 
following on the scale provided: 


PF 








The information I received was 


easy to understand when I... 


(1) Was offered suggestions by 


(2) 


€3) 


(4) 






(6) 


(7) 


(8) 


C10) 


C11) 


C12) 


C13) 


the system about what to 
do next 


Asked the system for more 
detail about a message 


Asked the system about my 
current situation and 
possible options 


Asked the system for help 
on a command(s) 


Asked the system for help 


about a subject area (e.g., 


FORTRAN or Inventory) 


Asked the system for an 
example 


Used a tutorial at the 
terminal 


Listed an index of 
available information 
resources at the terminal 


Read (part of) a document 
on the terminal 


Read (part of) a printed 
copy of a document 


Talked to another user 


Talked to a consultant or 
supervisor 


Other Cdescribe) 


| Strongly | 









agree 


| Strongly Didn't 


Agree Disagree disagree use 
















10) For those information resources 
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you used in the situation you described above, rate 


the following on the scale provided: 








| 
| 


The information I received 


(1) Was offered suggestions by | 


(2) Asked the system for more 


(3) Asked the system about my | | | 





(4) Asked the system for help | | | 


C13) Other (describe) 


Strongly 
agree 


Strongly | 
| Agree | Disagree | disagree | | 


was useful when I 











the system about what to | | 
do next | | 


detail about a message | | | | 


current situation and | 
possible options 





on a command(s) | | 


5) Asked the system for help 


| 
about a subject area (e.g., | | | ; 
FORTRAN or Inventory) ] | | 


(6) Asked the system for an | | i | 


example | 





€7) Used a tutorial at the 


terminal 


(8) Listed an index of 


available information 
resources at the terminal 


(9) Read (part of) a document 


on the terminal 


€10) Read Cpart of) a printed 


copy of a document 


€11) Talked to another user 


(12) Talked to a consultant or | | | | 


supervisor 


How long did it take you to obtain the information you needed? 


C1) 2 minutes or less 

(2) More than 2 minutes but not more than 10 minutes 

(3) More than 10 minutes but not more than 30 minutes 
(4) More than 30 minutes but not more than 60 minutes 
(5) More than 1 hour 

(6) Didn't obtain 
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C. GENERAL APPRAISAL 


In the following question we would like an overall evaluation of the information resources 
available on the system you were using. 


12) Rate the following on the scale provided: 






In general, the quality of 








the information I can obtain | | 
about my system is good | Strongly Strongly Not 
| when I... agree Agree Disagree disagree Available 


C1) Am offered suggestions | | 
| by the system about 
what to do next | | 


} (€2) Ask the system for more | | 
detail about a message 









Ask the system about 
my current situation | 
and possible options 





| (4) Ask the system for | | 1 | 
help on a command(s) | | | 1] 


(5) Ask the system for help | 
about a subject area | 
Ce.g., FORTRAN or | 
Inventory) | 


ne 
mine een 


Gh 


| (6) Ask the system for an | i| 
| example | | | 1 | 


(7) Use a tutorial at the | | | | 
terminal 








List an index of 
available information 
resources at the 
terminal 





(9) Read (part of) a 
document on the terminal | | | | 


| €10) Read Cpart of) a printed | 
| copy of a document | | 


} (11) Talk to another user | | | | 1 | | 






| €12) Talk to a consultant | 
| or supervisor | 







€13) Other Cdescribe) 





13) 


14) 
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What other types of information would you like to have available? Please list. 


r 
\ 











If you would like to make any additional comments about information resources for 
interactive systems, please do so here. 
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